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Section 4 

Laws of Large Numbers. 



Consider a r.v. X and sequence of r.v.s (X„)„>i on some probability space. We say that X„ converges to X 
in probability if for all e > 

lim P(|X„-X| >£) = Q. 

n — *oo 

We say that X„ converges to X almost surely or with, probability 1 if 

P(w : lim Xn{Lo) = X{aj)) = 1. 

Lemma 9 (Chebyshev^s inequality) If a r.v. X >0 then for t > 0, 
Proof. 

EX = EXl{X <t)+ EXl{X >t)> EXl{X >t)> tEl{X >t)= tF{X > t). 



Theorem 4 (Weak law of large numbers) Consider a sequence of r.v.s (Xj)j>i that are centered, EXi = 0, 
have finite second moments, EX^ < K < oo and are uncorrelated, EXiXj = 0,i ^ j. Then 



n 

i<n 



in probability. 

Proof. By Chebyshev's inequality we have 

n\Sn-0\>e) = nSl>e^)<^ 



-E{X, + ...+Xr,f=-^^EXf 



l^/,, ,,xO 1 \ ^ „ ,^2 ^ ^ ra— >oo Q 

i<n 



Example. Before we turn to a.s. convergence results, let us note that convergence in probability is 
weaker than a.s. convergence. For example, consider a probability space which is a circle of circumference 1 
with uniform measure on it. Consider a sequence of r.v. on this probability space defined by 

X,(x)=l(x4l+i + ... + i,l + ... + ^) modi). 



12 



Then Xk — > in probability, since for < e < 1 

n\Xk - 0| > e) = ^ ^ 

but, clearly, does not converge a.s. because the series J2k>i ^/^ diverges and, as a result, each point x on 
the sphere will fall into the above intervals infinitely many times, i.e. it will satisfy Xk{x) = 1 for infinitely 
many k. 

□ 

Lemma 10 Consider a sequence {pi)i>i such that Pi G [0,1). Then 

JJ(1 - Pi) =0 <^Y^pi = +0O. 

i>l i<l 

Proof. "^^=". Using that 1 — p < we get 

-Pi) < exp(-^pi) ^ as n ^ oo. 

i<n i<n 

"=>". We can assume that pi < ^ for z > to for large enough to because, otherwise, the series obviously 
diverges. Since I — p> e~^^ for p< 1/2 we have 



II (1-pi) >exp(-2 p,) 



m<i<n m<i<n 

and the result follows. 

□ 

Lemma 11 (Borel-Cantelli) Consider a sequence (^„)ti>i of events An £ A on probability space {il,A,F). 

Consider an event 

An io. := lim sup A„ := [J 

n>l m>n 

that An occur infinitely often. Then 

1- E„>iP(^n) < oo ^ P(^„ i.o.) = 0. 

2. If An are independent then X^„>iP(^n) = +oo P(A„ i.o.) = 1. 
Proof. 1. If Bn = Um>n then B„ D -Bn+i and by continuity of measure 

P(A„ i.o) = p( n = lim V{Bn). 

n>l 

We have 

P(B„) = P( U ^m) < X] ^(^n^) ^ as n ^ +oo because ^ P(A„) < oo. 

m>n m>n m>l 

2. We have 



v(n\Bn) = p(n\ IJ A„) =p( fi 

m>n m>n 

= A„)(by independence) = J] " P(^m)) = 0, 

by Lemma 10, since Em>„P(^m) = +oo. Therefore, P(B„) = 1 and P(A„ i.o) = p(n„>i -B„) = 1. 
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strong law of large numbers. The following simple observation will be useful. If a random variable 

X >0 then EX = V{X > x)dx. Indeed, 

POO pOO pX pOO pOO pOO 

EX= xdF{x) = / ldsdF{x) = / ldF{x)ds = / ¥{X > s)ds. 
Jo Ja Jo Jo J s Jo 

For X >{] such that EX < oo this implies 

POO 

Vp(X >i)< P(X > s)ds = EX < oo. 
i>i Jo 

Theorem 5 (Strong law of large numbers) IfE\X\ < oo and {Xi)i>i are i.i.d. copies of X then 

1 " 

— ^^Xj EXi almost surely (a.s.). 



n-. 
1=1 



Proof. The proof will proceed in several steps. 

1. First, without loss of generality we can assume that Xj > 0. Indeed, for signed r.v.s we can decompose 
Xi = X+ - X- where 

X+ = X,\{Xi > 0) and Xr = \X,\l{Xi < 0) 
and the general result would follow since 

1=1 i=l 1=1 

Thus, from now on we assume that X^ > 0. 

2. (Truncation) Next, we can replace Xj by 1^ = XjI(Xj < i) using Borel-Cantelli lemma. We have 

^P(Xi ^ Yi) = J2nXi > i) < EXi < oo 

i>l i>l 

and Borel-Cantelli lemma implies that P({Xj ^ Y^} i.o.) = 0. This means that for some (random) io and for 
i> io we have Xj = Yi and, therefore. 



^ n 1 ^ 

lim -VXi= lim -V^i. 

n— ^oo 77, ^ — ^ n— ^oo n — ^ 

i—1 

It remains to show that if T„ = X]r=i then ^ EX a.s. 



1=1 i=l 
— f jn,yi a.s. 

3. (Limit over subsequences) We will first prove this along the subsequences n{k) = [a*^] for a > 1. For 
any £ > 0, 



^p(|T„(fc)-Er„(,)|>£n(fc)) < ^-^Var(T„(fc)) = ^-^ 5] Var(r,) 

fe>l ^ ' i<n{k) j>l k:n(k)>i ^ ' 

< K'Y^EYI\ = k"^\ I x^dF{x) < K X {**) 
i>l * i>l * 

where F{x) is the law of X and a constant K = K{a) depends only on a. (*) follows from 

^ < n{k) = [a''\ < a'' 
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and if ko = mm{k : a'^ > i} then 



We can continue, 

(**) 



< 



^1 4 _ 4 K 

n(k)>i ^ ' a>'>i ^ <^ ' 



1 pm+l -I /•m+1 

E ^ E / -'dFi.) = E E ^ / -'^^(-) 

2>1 m<i'^'^ m>Oi>m '^'^ 
-I pm-\-l /'7n-\-l 

E / ^'^^(^) ^ E / ^^^(^) = EX < 00. 

m>0 m>0"'™ 



Thus, we proved that 



fc>i 

and Borel-CanteUi lemma implies that 

p(|T„(fc) - ET„(fe)| > en{k) i.o.) = 0. 
This means that for some (random) ko 

P(vfc > ko, |T„(fc) - ET„(fc)| < ffn(A;)) = 1. 
If we take a sequence = ^ > this implies that 

p(Vm> l,/c> A:o(m),|r„(fe)-Er„(;t)| < ^n(A:)) = 1 

and this proves that 

ET„ 



n{k) n{k) 



On the other hand, 

1 

i(fcj ^ ' n{k) 



ET„rt.^ = -^vv ^ EXiI(Xi < i) ^ EX as fc ^ oo, 



n,.., 

<n(fe) 



by Lebesgue's dominated convergence theorem. We proved that 



T 



(fe) 



. EX a.s. 

n{k) 



4. Finally, for j such that 



n{k+l) / 2 



n{k) < j < n{k + 1) = n{k) j—. — < n{k)a 

we can write 



n{k) j n{k + 1) 
and, therefore, 

1 T- T 

-^EX < liminf < limsup < a'^EX a.s. 
J J 

Taking a = 1 + m and letting m — > oo proves that limj_>oo = EX a.s. 
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